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Introduction 

At its best, data should be more than a 
number. It should tell stories. Measure 
capacity. Create, in a sense, a living 
picture in order to see the school and 
the system in a different way. Present 
the everyday in a precise and meaning- 
ful way. 

- A district administrator 

The Importance of Leadins Indicators 
in Education 

Improving student outcomes and closing achieve- 
ment gaps, both within a school and across a dis- 
trict, takes time - more time than is often allowed 
in typical large district and urban environments. 
Education leaders and community members need a 
way of examining their schools and school systems 
that allows them to understand when (and 
whether) progress is being made before the results 
show up in indicators like student test scores. 

Leading indicators - indicators that provide early 
signals of progress toward academic achievement - 
enable education leaders, especially at the central 
office level in a school district, to make more 
strategic and less reactive decisions about services 
and supports to improve student learning. These 
indicators are a way of viewing and using data to 
infonn systemwide decisions about education. 

This study builds on existing efforts by school dis- 
tricts to use data-informed decision making by devel- 
oping further the concept of “leading indicators.” 

The study described in this report focused on four 
districts that are at the forefront of the field in 
using data to inform decisions and examined how 
these districts are developing and using leading 
indicators for education. The study aimed to: 


• identify and describe the leading indicators 
used in four districts considered advanced in 
their use of data; 

• more broadly examine the infrastructure, 
resources, and supports each district has 
developed to support its use of data for deci- 
sion making; 

• look beyond student measures (though these 
are critically important) to other measurable 
areas, especially the effectiveness of central 
office practice; 

• bring representatives from these four districts 
together to review initial findings and help 
develop the work of the Annenberg Institute 
for School Reform at Brown University in the 
area of leading indicators and data-informed 
decision making. 

By describing how these four districts - Hamilton 
County (Chattanooga, Tennessee), Montgomery 
County (Maryland), Naperville (Illinois), and 
Philadelphia (Pennsylvania) - have developed 
and used leading indicators within the context of 
a strong district “data culture,” the Annenberg 
Institute hopes both to catalogue specific indica- 
tors that have been useful to these districts in 
increasing student achievement and to expand the 
notion of a leading indicator beyond easily identi- 
fied testing data to more difficult-to-measure but 
important measures such as student engagement 
and central office practice. 

Current Gaps in Research on Data- 
informed Decision Making in Districts 

Most of the research on data-informed decision 
making has focused on how school-based staff - 
teachers and principals - can use data in school- 
improvement processes. In a summary of the liter- 
ature, published separately (Supovitz 2008), we 
focused, instead, on data-informed decision mak- 
ing at the district central office level, because 


central offices are well positioned to make sys- 
temic changes and provide systemwide supports. 

The literature summary cited many studies that 
contain important insights into how data are used 
by school districts to inform decision making. 

Such studies provide reinforcing and supporting 
qualitative case study evidence that data-informed 
decision making plays an important role in district 
improvement strategies. This body of evidence is 
also suggestive of many of the important ways in 
which data-infonned decision making can be used 
to contribute to district improvement. 

However, the literature summary also pointed 
to several gaps in the literature related to how dis- 
tricts use data to inform decision making. 

• Most of the systems were heavily focused on 
analyzing student-performance data, and less 
so on other, process-oriented indicators of per- 
formance. While it certainly makes sense to 
focus on improving student performance, 
there are many indicators that contribute to 
student learning that are not direct measures 
of student performance itself. Bernhardt 

( 1998), for example, identified four categories 
of data for districts and schools: demographic 
data, instructional process data, perception 
data, and student-achievement data. She 
argued that selectively representing these cat- 
egories would produce a powerful system of 
indicators. 

• One of the biggest challenges that district 
leaders face is turning their data analysis 
into action. This theme emerges in several 
research studies at both the administrative 
and classroom levels. Administrators were 
uncertain how to apply the lessons from the 
data back to their practice, and teachers often 
reported that they struggled to turn data about 
student learning into instructional action. 

• Central office administrators are increasingly 
attending to providing feedback to educators 
about their school and classroom practices. 

But none of the research that we examined 
focused on the ways in which data were used 
by central office administrators to provide 


feedback about their own processes, systems, 
and performance. 

• While some of the research discussed opportu- 
nities for districts to synthesize the vast 
amounts of information they collect and iden- 
tify key indicators, there was little indication 
that much of this was going on in the field. 
Most of the data districts collect is focused on 
indicators of current student-performance lev- 
els. Little information is collected or analyzed 
on indicators that can serve as “signals of 
progress” - indicators that precede the even- 
tual achievement gains but tell us that those 
gains are likely to occur in the future. 

Our study was developed to get at these gaps by 
exploring the need for and use of leading indicators 
in education. 

Applying the Concept of 
Leading Indicators to Education 

The term leading indicators originated in economic 
theory (see sidebar). But it need not be exclusive 
to economics. In fact, leading indicators may be 
more useful in fields such as education or public 
health, in which growth is not necessarily cyclical, 
but where progress can be sustained over time. 

The challenge for such fields is to develop sets of 
indicators that not only reflect key investments, 
but also incorporate measures of important condi- 
tions that are known to be associated with 
improvement. 

Currently, the most widely accepted and used 
indicators in education are standardized-test 
scores. However, the manner in which standard- 
ized tests typically are utilized - given at the end 
of the school year and constructed as summative 
assessments - make them lagging indicators, like 
unemployment statistics. Scores on standardized 
tests, along with the other lagging indicators typi- 
cally collected and used in public school districts, 
usually arrive too late to help individual children 
or schools that are struggling. 

Lagging indicators confirm trends but do not eas- 
ily inform investments. Collecting information 
only on lagging indicators, as one of our study 
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informants told us, is like “playing the game with 
the scoreboard off. When the buzzer sounds at the 
end of the game, you flip the scoreboard on and 
say, Wait a minute. I thought we were ahead.’” 
These measures do not tell us whether the types of 
practices, people, strategies, materials, or tech- 
nologies school districts are investing in are likely 
to lead to higher student academic performance. 

Leading indicators, on the other hand, prioritize 
key areas that are particularly helpful in assessing 
progress toward goals. While educators do need to 
monitor lagging indicators, they also need leading 
indicators to help them see the direction their 
efforts are going in and to take corrective action as 
soon as possible. An indicator is a leading indica- 
tor when it is: 

• Timely and actionable: It is reported with 
enough time to change a course of action. 

• Benchmarked: Users understand what consti- 
tutes improvement on a leading indicator 
through construction of “metrics.” 

• Powerful: It offers targets for improvement and 
shows progress - or a lack of progress - toward 
a desired outcome before that outcome occurs. 

Our study districts associated lagging indicators 
primarily with student achievement at the end of 
each grade (as measured by letter grades and stan- 
dardized tests) and high school completion (or 
dropout). We would like to note, however, that an 
indicator that is considered leading in one context 
can also be considered lagging in a different con- 
text. It is common knowledge that education out- 
comes are connected to one another. For example, 
there is a well-established connection between 
early reading proficiency and later outcomes, from 
upper elementary school to secondary school, 
which is one of the reasons, as will be seen later in 
this report, that early reading proficiency rose to 
the top as a leading indicator in our four study dis- 
tricts. With a change in perspective, however, 
early reading proficiency can easily be seen as an 
outcome, or lagging indicator, of data collected at 
earlier points, such as letter identification and oral 
vocabulary in kindergarten. The leading indicators 
we identify could, for the most part, be seen as lag- 
ging indicators when used in a different context. 


LEADING INDICATORS IN ECONOMICS 

y 

Economists do not wait for unemployment rates to 
be released to see if the economy is on the road 
toward full employment. To determine if employ- 
ment rates are likely to rise in the future, they examine 
certain sets of data, such as factory orders, known as 
leading indicators. 

The Conference Board, a nonprofit organization, uses 
three indexes, each composed of several indicators 
(monitored by various government agencies), to 
determine the economic and fiscal standing of the 
United States : 

Leading index. A monthly index that predicts the 
direction of the economy’s movements in the 
months to come. The index is made up of ten eco- 
nomic indicators whose changes tend to precede 
changes in the overall economy. 

Coincident Index. An index composed of four 
cyclical economic data sets that provide broad- 
based measurements of current economic condi- 
tions, helping economists and investors to determine 
which phase of the business cycle the economy is 
currently experiencing. 

Lagging Index. A monthly index that confirms the 
direction of the U.S. economy’s movements in past 
months. The index is made of up seven economic 
components whose changes tend to come after 
changes in the overall economy. 

These indexes are produced regularly and reported 
widely. Despite their imperfections, leading eco- 
nomic indicators play a critical and effective role in 
strategic planning in all sectors of the economy and 
its regulation. 

Definitions adapted from lnvestopedia.com. 
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About the Study 

To further explore how leading indicators might 
work in practice, we sought out four districts that 
were at the forefront of using data-informed deci- 
sion making. We identified these districts by 
reviewing studies focusing on that topic and not- 
ing sites that were mentioned, speaking to col- 
leagues who are knowledgeable about the topic, 
and drawing on our own experience. 

We narrowed down an initial list of over fifty dis- 
tricts to twelve that were cited multiple times. 

We interviewed one district representative in each 
of those twelve districts and selected four districts 
- Hamilton County (Chattanooga, Tennessee), 
Montgomery County (Maryland), Naperville 
(Illinois), and Philadelphia (Pennsylvania) - to 
be part of a cross-case analysis focused on leading 
indicators. The research involved interviews with 
district and school leaders in the four communi- 
ties. (See the Appendix for research protocols.) 

The Leading Indicators 

W e learned of efforts to develop and use what 
we would call leading indicators in all four of 
our study districts. Figure i lists these indicators 
and describes the interventions our study districts 
undertook as a result of assessing these indicators 
and how broadly the intervention was applied. 
Each of the leading indicators in Figure i is 
described in more detail in this section. 

The indicators fall into two groups: 

• Indicators that the districts have the most 
information on. These indicators are com- 
monly collected by most school districts; they 
have simply been prioritized in our study dis- 
tricts. 

• Indicators that are not as commonly col- 
lected, about which our study districts have 
some information. Not coincidentally, these 
are apt to be harder-to-measure, less-quantifi- 
able constructs such as “teacher quality.” 


Common Indicators 

All of our districts had ways to monitor student 
learning, as described in the section of this report 
entitled The Role of the Central Office. But, in 
addition, they tracked other indicators they felt 
were critical to student achievement, such as early 
reading proficiency and enrollment in pre-algebra 
and algebra, among others. All of the indicators 
described here were examined at varying levels of 
aggregation: by individual student, by classroom, 
by school, by subgroups such as race/ethnicity, and 
for the whole system. 

Early Reading Proficiency 

Early reading proficiency was the most common 
leading indicator examined by our study districts. 
It was often the first thing district leaders and 
partners mentioned when asked if they could 
identify any high-leverage indicators. A central 
office administrator told us: 

If we start looking at where our children are in 
third-grade reading, which we’ve seen articles 
and documents written that if a child is on 
grade level ... by grade three they’re on a very 
good path for success. [The superintendent] has 
put millions of dollars into our early-childhood 
education program. . . . We have established 
benchmarks in kindergarten and in grade one 
and in grade two for on-grade level reading of 
text. 

To one external partner, it was as simple as: 

“We know that third-grade reading scores are 
an important predictor of getting through high 
school.” To address this leading indicator, our 
study districts invested in early childhood educa- 
tion to increase the numbers of students meeting 
this benchmark and provided interventions, such 
as tutoring and double doses of reading instruction 
to individual students not reaching the bench- 
mark. 
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Enrollment in Pre-algebra and Algebra 

A central office administrator told us: 

We use data K to 12. We have different lever- 
age points - targets - we think will move the 
bar for our kids, increase performance. One 
leverage point is how many kids we have in 
sixth-grade math in fifth grade. We want every- 
one to complete algebra by grade 8 . We want 
them to complete algebra by seventh grade if 
they’re on the fast track. 

The other three study districts also had some kind 
of systemwide initiative focused on helping stu- 
dents get ready for algebra and to increase enroll- 
ment in algebra. The ultimate goal was to help 
students master algebra sooner in their academic 


careers. They monitored enrollment and perform- 
ance in math classes. As one partner told us in 
response to a question about important indicators: 

The level of math courses students take and 
their performance in math. Math is a huge 
stumbling block for kids’ success in high school 
and beyond. 

Over-Age/Under-Credited Students 

Two of our study districts work to identify students 
who are “over age.” In high school, that typically 
might mean students who have only accumulated 
enough credits to qualify as a sophomore but are 
actually old enough to be a junior or a senior. In 
elementary school, over-age students are those 
who are a year or more older than their peers in 


Fisure 1 . 

Leading indicators used in our study districts and their associated interventions in the same districts 


Leading indicator 

Associated intervention(s) by our study districts 

Level applied to: 

Early reading proficiency 

Reading interventions (double dose, Reading First, tutoring, etc.) 

Individual student 

Investment in early childhood education 

System 

Enrollment in pre-algebra and 
algebra 

Provide math tutoring and other supports 

Individual student 

Increase enrollment & course offerings 

System 

Overage/under-credited 

Alert someone at the school about students meeting these criteria 

Individual student 

Establish transition goals 

School 

Create grades 6-1 2 academy to reduce transitions 

System 

College admission test scores 

Change placement and provide support to succeed in more 
rigorous courses 

Individual student 

Attendance and suspensions 

Intervene with student and parents 

Individual student 

Adopt strategies to reduce violence and disruption 

School 

Special education enrollment 

Reduce number of separate placements 

System 

Inclusion 

Student engagement 

Benchmark and look at data 

Classroom or school 

Develop rubrics 

Teacher and principal quality 

New teacher evaluation 

School or system 

Coaching for teachers and principals 

Conversations about data 
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the same grade. Concern about a lagging indicator 
- dropout rates - was the impetus for this focus in 
one district. 

We started looking at dropout rates. What are 
some of the factors leading up to it? We did a 
little research and found that 64 percent of 
dropouts are over age [for the grade they are 
in]. So we put an indicator at every grade that 
alerts schools that kids are over age. 

We are trying to avoid having sixteen-year-old 
ninth-graders. [But] you can’t just wait to high 
school to address it, so we brought it down to 
kindergarten, try to provide support to at-risk 
kids. ... Is the child over age; has poor atten- 
dance; shows some at-risk factors? Then we 
intervene earlier. We are trying to be more 
proactive and preventive. 

- Central office administrator 

One district, in particular, focused on data around 
student transitions, especially from fifth to sixth 
grades, eighth to ninth, and ninth to tenth. A dis- 
trict partner talked about the central office’s 
“change leadership group,” and about how in look- 
ing at district data, they found that: 

Lots of students were not successful in sixth 
grade. There’s lots of retention, and a huge 
number of retentions in sixth grade, as com- 
pared to fifth grade. It’s a transition problem. 

The number of disciplinary and special education 
student referrals was also much higher in sixth 
grade than in fifth. Now, in addition to literacy 
and math goals, the central office has established a 
“transition goal” between elementary and middle 
school. 

Transition goals have also been established 
between middle school and high school. As the 
district partner put it, the “overarching goal of the 
entire initiative is to prepare every single middle 
school student . . . for a rigorous high school cur- 
riculum.” According to a district partner, 

In 2001, schools were reporting 100 percent 
promotion. But the more you poked at it, [you 
learned] every high school had a different way 


to determine what made a ninth-grader a 
tenth-grader. In some schools, if you showed up 
the next year, you were marked as a tenth- 
grader. It took us forever to go back and change 
board policy. . . . You’ve got to get the data 
clean and clear. It took us two years before we 
could get precise numbers. 

Now, “every academy [talks] about ninth-to-tenth- 
grade promotion.” And not only are schools talk- 
ing about middle school to high school transition, 
but they have also used data on over-age and 
under-credited eighth- and ninth-graders to create 
a 6-12 academy that focuses on this group of stu- 
dents. A central office administrator said that in 
this school, they started an eighth-grade transi- 
tional academy with a focus on math, literacy, and 
physical science. 

It’s designed to help this [over-age and under- 
credited] group of students make a smoother 
transition [to high school]. It’s easier there 
because of the 6-12 configuration. Have we 
developed all options for this group? No, but 
it’s come up out of our data. 

College Admission Test Scores to Clarify High 
School Placements 

All of our study districts sought out ways to assure 
that high school students were appropriately 
placed, either by assessing students’ credit accumu- 
lation compared with their age, as described in the 
section Over- Age/Under-Credited Students, or by 
examining college admission test scores. Two dis- 
tricts in our study have examined scores on col- 
lege-entrance examinations (such as the SAT and 
the ACT) and their associated preparatory tests 
(e.g., the PSAT) and curricula (e.g., ACT Plan and 
Explore). They identify students who score high 
but are not enrolled in advanced courses or who 
are in danger of dropping out. 

For example, we had students who were drop- 
ping out of high school who were very smart. 
We didn’t realize until we looked at the data. 
We found students who scored 1 9 or above on 
the ACT. These students need to be supported. 

- Central office administrator 
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The PSAT, to me, is the most powerful [indi- 
cator], . . . We’ve defined cut scores ... so 
there’s now a tool that schools have access to 
through FileMaker Pro. With that, we pull up 
a kid’s [record] and we can identify what kids 
have those scores that are not currently 
enrolled in honors or AP. 

- Central office administrator 

So we analyzed the PSAT to see if there was 
any correlation between courses and their 
score. Did it have an impact on the [student’s] 
SAT [score by] just taking the PSAT? And we 
found out that if you take the PSAT you score 
better on the SAT. We also knew that if you 
get more than 42 or 43 on the math or English 
[PSAT], you could be in a highly rigorous 
course. Research from the College Board sub- 
stantiated that. . . . We made it so that if you 
score a certain level, kids have to be in the 
[more rigorous] course. If they’re not in there, 
you have to put them in. Have to put the kids 
scoring high in these courses. But they need to 
have the supports if they haven’t been in 
higher-level courses in the past. 

- Central office administrator 

One district is piloting the ACT’s eighth- and 
tenth-grade college- and career-planning tests. An 
external partner noted: 

Right now, all eighth- and tenth-graders are 
taking Plan and Explore for ACT. How do you 
backwards-map what you need to know in 
sixth grade to get 22 on ACT for a college 
scholarship from the state ? How do you struc- 
ture curriculum and coursework to accomplish 
that? 

This district is also utilizing a Web site that corre- 
lates state assessment scores to predict ACT scores 
and expected salary figures for future employment. 
Another district enrolls students, particularly stu- 
dents of color, in AP courses if they score high on 
standardized tests. 

Student Attendance and Suspensions 

Districts have made headway collecting and shar- 
ing school- and district-level attendance rates 


with greater frequency. In one district, previously, 
attendance -data reports were delivered to schools 
once a month and at the end of each semester. 
Now, attendance data are shared on a ten-day 
cycle, allowing for principals to make necessary 
changes. According to one principal, “I have the 
central office get me grade-level attendance data 
because my ninth and tenth grade have awful 
attendance. I’m missing 15 percent of my kids.” 

In the same district, a school leadership team 
asked the district’s data team to present to the fac- 
ulty on the relationship between attendance and 
student achievement. Important data in this effort 
are suspension and major incident rates. The key 
is to look not just at the overall percentages, but 
also at whether it tends to be the same students 
who are chronically suspended - and to build a 
subsequent understanding of how many instruc- 
tional hours these students are missing and the 
academic costs of those absences. One district 
leader put it this way: 

The serious incidents and suspension indica- 
tors were connected to the theory that we all 
believe in - that if you have a highly volatile 
school, you can’t have really good instruction 
take place. [So we] help teachers and principals 
monitor and bring down the level of violence 
and disruption. 

Harder-to-Quantify Indicators 

All of the indicators described above are fairly 
easy to measure and have, for the most part, been 
collected by most districts for years. But there were 
also some indicators that our districts examined 
that were more difficult to quantify and are not 
collected widely by most school districts. These 
indicators include special education enrollment, 
student engagement, and teacher and principal 
quality, including teacher turnover. 

Special Education Enrollment 

Under No Child Left Behind, special education 
students receive a great deal of attention due to 
the need to make adequate yearly progress with all 
subgroups. As one central office administrator put 
it, “It’s no longer compliance with ‘dotting i’s.’ It’s 
now achievement and outcomes for all students.” 
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All of the districts in this study tracked data on 
special education students, though sometimes 
the data were not integrated. For example, inter- 
viewees in one district mentioned that informa- 
tion in special education students’ IEPs existed 
only on paper, not in the district’s data warehouse, 
and integrating this data was a priority. 

A central office administrator in one district 
specifically mentioned using data to make changes 
in special education programs. This district had 
created a number of schools and centers for special 
education students. These students were removed 
from mainstream classes to attend these schools 
and centers. However, as the administrator put it, 
We have a lot of data now that shows that 
[these special schools and centers have] not 
been successful for many of these students. . . . 
So we’re now trying to use that data to help us 
bring those kids back into the mainstream, the 
regular schools and regular classrooms through 
inclusion and other practices and taking those 
resources that we had out of those special cen- 
ters and schools and putting them into the reg- 
ular schools and regular centers to support 
those kids. So we’ve been using data to show 
that and try to drive that. 

This decision has upset some parents, who wanted 
their children to remain in these schools and cen- 
ters and believed the regular schools would not be 
as supportive for their children: 

But we’ve just been continually focusing on 
that data and as you look at that data ... a lot 
of those kids that were in those special place- 
ments are African American. . . . Taking 
African American students and identifying 
African American students and putting them 
into separate places has just not been success- 
ful. So using data to help drive some of the 
strategies in our strategic planning, eliminat- 
ing some of these special centers, eliminating 
an evening high school program where stu- 
dents haven’t been successful - taking those 
kids and putting them back into the regular 
school. 


When fifty or sixty upset parents picketed outside 
of a board meeting, the administrator again 
pointed to the data: 

They’re issues that raise a lot of emotions in 
people. . . . The only reason we’re able to make 
any of these changes is because of the data. 
And it’s more than just telling a story. It’s really 
having good hard data to show that kids 
haven’t been successful in the other place- 
ments, in these other settings. And having the 
data when they are included to show that it’s 
making a difference for the kids is helping us 
at the district level to be able to support strate- 
gic planning to implement strategies that we 
think are going to be necessary in order to help 
all kids be successful. To monitor that, to see 
that it is making a difference and, if it isn’t, 
then to use the data to help us do something a 
little bit differently. 

What we try to inform people about is when 
you get the data and it shows that you’re 
not attaining the results that you wanted, it 
doesn’t mean that you kill the program. It just 
means that you need to identify what’s not 
working and improve it. Part of the challenge 
with certain elected officials is that they 
believe that the data should be used to see 
whether you fund a program or not fund a 
program, as opposed to using it to make 
improvements that you need in that program. 

Student Engagement 

Several districts used different measures - includ- 
ing student surveys and classroom walk-throughs 
- to address the issue of student engagement at the 
school level. For example, one central office 
administrator explained the district’s use of “focus 
walks”: 

We do a thing called focus walk, and we have 
teams of staff members go into a building and, 
basically, they peek their head into a classroom 
for a few seconds and they look at the activities 
that are going on in the classroom and they 
rank how students are engaged in the class- 
room. Everything from passively sitting there 
and being lectured to, to taking control of their 
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own learning and doing activities that are 
helping create their own meaning from what 
they’re doing. And we gather [and look at] that 
data. . . . We benchmarked a couple of years 
ago, and so now we’re looking at the data and 
in terms of best practices and how we want stu- 
dents engaged in learning, what do we look 
like, what do we sound like in terms of growing 
our schools. 

A district principal agreed, saying that: 

Teachers have gone out of their way to develop 
rubrics, gathering that kind of data, which 
increases student engagement and also should 
increase student success. 

Districts also reported that they did frequent stu- 
dent surveys on topics ranging from technology 
use to students’ social-emotional needs. An 
administrator from one district said, 

So we survey all of the students and ask them 
everything from, “Do you ever feel bullied at 
school? Do you feel welcome at school?” to find 
out the social-emotional status of the school. 

Another central office administrator pointed out, 
“So we do a lot of internal surveys, climate surveys 
of schools, senior exit surveys, the climate series of 
parents, students, and staff.” 

However, some admitted that student engagement 
is not easy to quantify. One central office adminis- 
trator said, 

I believe our students on task is certainly a cor- 
relational behavior, but we have struggled with 
some of the things that we think are good key 
performance indicators, getting them to a point 
in which they roll up and can be quantified and 
used in a format such as this. 

Thus, there’s a belief that student engagement, as 
defined in a myriad of ways (e.g., school climate, 
time on task) is important, but there are only lim- 
ited ways to easily measure and aggregate student 
engagement data so they become useful for admin- 
istrators and teachers. Several of our districts 
expressed an interest in finding better measures of 
their students’ well-being but had found limited 
ways to get to this data. Participants from one dis- 


trict in particular almost universally commented 
on its importance. The superintendent said, 

We have been struggling with issues around 
diversity, how to tackle it. Interesting question 
across the district: kids in this school are toler- 
ant of kids different from them on the survey. 
The number was still high, but there was a drop 
from last time. What can we do at the district 
level to modify what we are trying to do with 
social-emotional learning? 

A principal in that district added, 

If a kid feels valued in the system, he is going to 
learn better. Right now, it is hard for a teacher 
to get [that information]. We have to survey 
[the kids]. Probably each teacher is doing the 
same survey in each class. We might be able to 
provide that for him or her. 

A central office administrator summed it up best, 
saying: 

How to assess social-emotional data is an area 
where we tend to go by gut rather than data. 
We need training on what tools are out there, 
what really is going to inform how we help kids 
in that area. Lots of research shows that social- 
emotional concerns can affect achievement. 
We do have some; we introduced the Manners 
Matrix and are trying to tweak [it] with social- 
emotional learning goals, and a school percep- 
tions survey [was] completed recently. [We] got 
some; we need to collect in a systematic way 
that will inform our decisions around the 
social-emotional piece. 

Efforts to understand student engagement that 
were described earlier are nascent attempts at get- 
ting at the broader construct of students’ social 
and emotional development. 

Teacher and Principal Quality 

Districts have started thinking about measuring 
teacher quality in instruction, though it is still at 
an embryonic stage in most of them. At the most 
basic level, 

For me, I look at teacher turnover every year, 
that’s a big piece for me, looking at trying to 
hire twenty -five middle and high school teach- 
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ers. The goal is to reduce that turnover num- 
ber. So it’s a critical piece of data. I need teach- 
ers who are a good fit for my building. A large 
piece is determining whether these teachers 
work with the students in my school. 

Coaching is a major strategy for developing the 
skills of both new and experienced teachers. As a 
central office administrator described, 

When you’re a good coach, you’re coaching 
people’s technique. People out here go pay lots 
of money to get their golf swings coached, so 
what’s the matter with coaching teachers? 
You’re having somebody there examine the 
swing and help them get better. You turn 
upside-down the whole idea of learning and 
you make a whole bunch of learners. We’re all 
in this together and you have observers and 
coaches and idea banks and you create councils 
on teaching and learning that we create in 
every one of our buildings. You create embed- 
ded staff developers that aren’t evaluators. You 
create unions and support systems. You create 
all kinds of ways that create mentor/coaching 
relationships. 

The difficulty with this kind of coaching tech- 
nique is the labor-intensive process of collecting 
data and using that information to train teachers 
to be more effective. The administrator also 
described the district’s development of a database 
to track all of the different variables that go into 
teacher effectiveness. As he described, 

Well, there’s a database on how you use time. 
There’s a database on how you measure quality. 
There’s all of these databases on teacher abili- 
ties and it’s a relational database of what you’re 
trying to match all of these things, the Rubik’s 
Cube. Did I get the right teacher with the right 
student with the right amount of time with the 
right skill set and interventions that will cause 
this student’s perfonnance on these indicators 
that we have agreed upon to go up? Does the 
student know and be able to keep track of that 
and does the teacher know and be able to keep 
track of that? And can we engage anybody 
from the outside to know and keep track of 
that? 


Like collecting data on student engagement, get- 
ting easily quantifiable and usable data on teacher 
and principal quality is complex and difficult. Fur- 
thermore, as the administrator pointed out, the 
data system itself may not be the crucial factor in 
improving practice, but rather the conversations 
that arise from that data: 

How do we put that data out there in such a 
way that causes us to have a conversation 
about what’s right, what’s wrong, where’s the 
variance? Because what you’re doing all the 
time is, you’re trying to look at the variance. 
Anytime you introduce [multiple] schools, 
you’ve got a lot of variability. Then within 
those schools, you’ve got variability, and within 
the class age range, you’ve got variability. 

So what you’re always trying to do is to find 
those most efficient and effective practices that 
actually engage the learner and cause the 
learner to want to participate. You know, the 
right learning culture - you can’t embed this in 
a bad learning culture. And cause the adult 
learner to want to learn more about the learner 
and cause them both to look at these objectives 
that they want to hit or these targets or stan- 
dards or whatever you want to call them. 

And how do you facilitate that? So you’re 
always facilitating a culture and time and all of 
these other things simultaneously and we learn 
from Citystat or Comstat or whatever Bratten 
called it. We learned that there are ways to 
depict this type of data that cause conversa- 
tions, but to send the data out doesn’t do any 
good. It’s the conversation that does the good. 

In another district, a teacher evaluation has been 
implemented, after three years of discussion and 
planning. They also reexamined surveys on 
teacher satisfaction to determine whether that had 
any impact on student achievement. Another sur- 
vey of teachers and administrators showed that 
supervisor ratings were meaningful to teachers. A 
central office administrator said, “We want to 
know whether we are giving the quality feedback 
to a teacher or . . . saying, after so many years, 
‘you’re an excellent teacher.’” Additionally, the 
district has implemented an interview tool that 
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scores teacher applicants. The district plans to 
determine whether this tool is “actually sorting 
out who are the best teachers.” The district also 
tracks teacher professional development and 
teachers who are released and has developed an 
exit survey with the collaboration of the union. 

In a related effort, the districts were particularly 
interested in gathering additional data about 
teacher preparation and training. For example, 
one district’s vendor said, 

I also wonder how good universities are doing 
with teacher preparation for training teachers 
on how to use data. I doubt [the local univer- 
sity] has very much of this. [A data specialist] is 
invited once in a while to speak to students, 
but other than that, I don’t know. How do we 
help our teacher-preparation programs and the 
universities prepare our teachers better to enter 
a data-driven system? 

A central office administrator from the same dis- 
trict said, 

Another area we didn’t talk about is K-16 
- connecting with colleges around matricula- 
tion, training teachers. Going to the schools, 
sharing information, talking with teachers, and 
realizing in every building there is something 
you can learn. 

Another central office administrator, commenting 
on the changing nature of the economy, said, 

The president of [the local university] system 
said in a presentation recently that if you take 
a student coming out of school today, he or she 
may have eighteen career changes. If that’s 
what we’re looking at, then the future is now. 
You can’t prepare students for that type of 
future if you give ... a teacher a piece of chalk 
and close the classroom door. We need to bring 
community organizations and businesses into 
the conversations. 

Similarly, several central office administrators 
were interested in gathering additional informa- 
tion from universities about student teachers and 
teachers coming from their programs: 

I would like to gather data from student teach- 
ers. Talk to supervisors. What universities and 


colleges are they coming from? Are there areas 
where they are lacking? Areas where they 
excel? 

[I would like] more data on the teacher. For 
example, what college they attended. Is it pos- 
sible to use school code like ACT does? Once 
we get that electronically, we can do more with 
the teacher piece. 

In the end, we heard much focus on principal and 
teacher training and quality, but less emphasis on 
how to measure those variables in a systemwide 
fashion. There are emerging strategies, however: 
the coaching model, facilitating conversations 
about teacher quality, and looking at easily meas- 
ured variables like teacher turnover. Creating a 
robust system to look at teacher and principal 
quality, however, is only just emerging. 

The Need for Better Lagging 
Indicators 

L ike most districts, the four districts in our study 
collect a lot of data. But there are still areas 
where more information is required. We’ve high- 
lighted a few of these above. Harder-to measure 
constructs like teacher quality and student engage- 
ment require easier-to-gather and more reliable 
measures. 

But it’s not only the leading indicators that can 
be lacking. Our districts also indicated that data 
about outcomes - the lagging indicators they have 
access to - are also inadequate. What we learned 
from our study districts is that a sound data- 
informed decision-making system needs robust 
leading and lagging indicators. Currently, districts 
typically have data on graduation and dropout 
rates, test scores, and grades, but they need to 
know more about other kinds of lagging indicators 
like post-secondary enrollment, job placement, 
and life satisfaction. 

For example, all of our districts were able to 
minutely dissect student outcomes through the 
twelfth grade. But as soon as their students gradu- 
ated, they had limited ways to track them. The 
ultimate proof of the education that districts 
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provide is not the students’ scores on standardized 
tests or their grades, but their success after high 
school ends, in college or the world of work. It was 
extremely difficult for our districts to know what 
happened to their graduates. 

Right now, one thing that’s so frustrating is 
how little information is coming from colleges. 
We have a working group to meet with the 
local college presidents to look at data. Across 
the country, there is just an appalling college 
graduation rate. Forty-five percent of kids grad- 
uate in six years at the [state university]. We 
don’t know what correlation there is between 
grades, are there courses they need to take in 
high school, or the impact of ACT scores. If 
we’re really talking K-16, why can’t we pick a 
hundred kids from [the district], look at their 
transcripts, and begin to see paths to success? I 
want to know from each school. We just don’t 
know what the recipe is for college success. 

Two of our study districts were working in partner- 
ship with the National Student Clearinghouse to 
get some information about post-secondary college 
enrollment, but the data are limited and expen- 
sive. Informants in our study districts desired 
more systematic and easier-to-access data that 
would help them understand their students’ post- 
secondary outcomes. 

Using Indicators to Target 
Supports to Individual 
Students 

T he four districts we studied were very different 
in terms of size, student and community demo- 
graphics, and student outcomes. But there was a 
remarkable similarity in the reasons they gave for 
developing a data-informed decision-making sys- 
tem and in describing the future focus of those 
efforts. In this section, we describe their rationale 
for using data - differentiation and action - and 
their nascent work in understanding what they 
called “student trajectories.” 


The Qoals: Differentiation and Action 

One of the fundamental flaws is people never 
figure out where they’re going. And when 
they’re building data systems, all they’re trying 
to do is keep track of all the stuff that they’re 
doing. And so if I were to give some advice 
to anybody, it’s, What’s your target? And I see 
so many data systems that are just that - data 
systems - and that means a systematic gather- 
ing of something. For what? 

- Central office administrator 

While our respondents talked about a number of 
reasons for having a data-informed decision-mak- 
ing system, the two overarching themes of their 
efforts were differentiation and action. They felt that 
a robust data system, manipulated appropriately, 
could give them information about the needs of 
specific schools, students, and, in some cases, 
teachers and could point toward actions that 
could be taken to improve outcomes for students. 
As a central office administrator said succinctly, 
Our goal is to differentiate instruction for our 
students, to give teachers tools to be more 
agile, and to free up time from doing and scor- 
ing, and placing kids. 

Another central office administrator described this 
emphasis on understanding each child and acting 
on that understanding: 

So, if one of our standards is a student’s ability 
to make connections to real-world problems to 
what they read - whatever it is - you dig down, 
and it’s not only what are we doing as a school, 
what are we doing as a grade level, or what are 
we doing as a classroom, but who are the kids 
that are getting it, who are the kids who are not 
getting it. 

We’ve kind of had this mantra of, “It’s each 
child” and really put the emphasis on “Every 
child learns,” maybe a little bit differently, but 
how do you get down to each child, and just 
that kind of philosophy of thinking, “It’s not 
every child, it’s not my classroom of children, 
but every child is a little bit different.” And the 
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idea is to easily present and access infonnation 
. . . the most efficiently and effectively so we 
can affect that child’s learning. 

These three respondents also articulated this 
focus: 

Across the board, the big curricular instruc- 
tional change is a focus on differentiating 
instruction. Taking every kid and moving them 
just a little. 

- Teacher 

If you go into a school, I think from just three 
or four years ago, the practice has changed 
so dramatically. ... It truly is helping teachers 
get down to the individual child. I think often 
in the past we just had the groups of kids, so we 
would look at if African American kids weren’t 
doing well or if poor kids weren’t doing 
well. Now . . . what I see is getting down to the 
individual student and really beginning to 
focus on putting in place strategies and inter- 
ventions to help those individual students. So 
I really think that teachers now have the data 
and they’re using data to drive that individu- 
alized instruction. 

- Central office administrator 

One thing we really focused on came from a 
conference I attended: we have to start talking 
about each student instead of every student. 
Data is the answer to that; when looking at the 
gross level, we miss some of the detail, [such as] 
special education kids not achieving as well. 
Data analysis makes sure we provide educa- 
tional opportunities for each child. 

- Central office administrator 

Another central office administrator used a med- 
ical analogy when describing his district’s use of 
data for differentiation and action: 

So we can drill down and find out what is the 
issue after we take the “x-ray” at a child level, 
a classroom level, a building level. Then that 
triggers an alarm somewhere about what to do. 


Later in the same interview, he continued the 
analogy, describing the data produced as a kind of 
“diagnosis”: 

And the more time you spend on diagnostics 
and knowing what your trajectory is, the 
better [the impact is] you’re going to have with 
your treatment, because you can target your 
treatment. 

One respondent differentiated the work in his 
school system, compared with most school dis- 
tricts: 

I’m telling you, most schools in America are 
stuck in Phase I, [which is] “data as informa- 
tion.” You’ve got to go farther and say, “So 
what? Now what? What does it mean?” . . . 
Okay, you’ve got the information. You’ve got 
all this stuff. What kind of knowledge are you 
putting it in and then how do you take that 
and move that to action? 

Understanding Student Trajectories 

This emphasis on differentiation led three of our 
four study districts to use statistical modeling to 
identify particular trajectories and pathways for 
students, based on multiple indicators. They had 
already successfully prioritized a set of indicators as 
described in the leading indicators section. Their 
next step was to synthesize these and other indica- 
tors to create even more precise and accurate por- 
traits of their students and schools. The models 
the three districts were developing are designed 
predicted future student performance and could 
provide opportunities for educators to intervene 
before negative outcomes occur. As one central 
office administrator told us, 

We’re looking desperately at what are those 
indicators that are predictors. And what are 
those ways of measuring them that are not so 
intrusive, yet tell a story that doesn’t consume 
a great deal of time and gets you feedback 
quick. 

In one district, understanding students’ trajecto- 
ries, particularly whether students are on track for 
graduation, started as early as kindergarten: 

What we did was take AP Calculus and we 
took AP English, that’s your math and your 
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language arts, and said, Where did you have to 
be ... in grade two to get [to AP Calculus and 
AP English] ? . . . Then we looked at the tra- 
jectory. ... If they’re on track here, is there a 
high probability or correlation [with the next 
point] ? We did a study to predict that if you’re 
on track [at a particular point], would you be 
able to do this? Then we found out there was a 
strong positive correlation [between] second- 
or third- [grade reading proficiency] and high 
school [achievement]. 

And so now we’re trying to figure out how to 
measure an individual student’s performance 
along this trajectory. You, for example, and 
you’re in the fifth grade, where are you at? . . . 
Are you above the line of trajectory or are you 
below the line of trajectory? 

Supervisors of principals in the same district use 
the data from the trajectory in discussing school 
performance: 

We, as supervisors of principals, will go into 
schools on regular visits and we will directly 
talk about, according to level, how schools are 
doing with those points, but we will come in 
with data. Principals have data. 

We will sit down and we’ll analyze, where are 
you with middle school math completion in 
elementary school? Who’s in these classes? 
How are you preparing kids to get to that point 
over a five-year period from kindergarten 
through grade four? Who’s in the classes? 
Who’s teaching and what’s the certification? 
Are you making progress? What do you project 
for the future for all the subgroups? So, we’ll 
have data, and they have data, and we’ll talk 
in depth about those things. 

Another district’s respondents described their 
beginning work on using statistical modeling to 
understand typical student pathways, including 
critical stepping-stones that can determine 
whether they are on track for success. They 
referred to this effort as a growth model: 

Part of what we’re also building into the ware- 
house is a growth model, so we can better look 
at student data trajectory and student growth - 


what’s expected, who’s above, who’s below - 
and earlier find and target students who don’t 
look like they’re going to be on an expected 
path of academic growth. That’s really this big 
piece kind of pulling this all together that’s 
driving the implementation of that system at 
the moment. 

An external partner working with this district 

focused on what such efforts would produce: 

[If you have] an eighth-grader scoring at this 
level, give them the path. This is where they 
will be as sophomores in high school. You can 
expect them to be at this point. You can see the 
trajectories, these pathways. [We’re] talking 
about . . . the ways [students] can move and 
what we [as educators] can do [to improve out- 
comes]. 

A third district not only examined the pathway of 

a successful high school graduate, but also tried to 

define what helped them achieve: 

[We asked,] What does an accomplished 
twelfth-grader look like? Then, internally, we 
looked at what five or six things that if we did 
right will actually more likely create that to 
happen. . . . One of them was workforce excel- 
lence and starting earlier. 

So, after examining six years of data on prior 
graduates, we found out that there were some 
things that got them more work ready and col- 
lege ready, and we followed them by individual 
social security number and identifier and got a 
special latitude from the state to do that. And 
we found out that kids [who] had certain levels 
of preparation actually earned more money, got 
out of college quicker, and those kinds of 
things. 

So then it became, how do you back-map that, 
and are there predictors that you can pick up 
early on so when the train gets off the track, 
you can start doing something about it. We 
found out one of the major predictors was great 
teachers, and we found out that one of the 
major predictors of having a group of great 
teachers was a [great] principal. So we just 
started putting it all together. 
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The hope in these districts was that understanding 
various student trajectories and what contributed 
to them would help identify students who were in 
danger of failing, dropping out, or otherwise under- 
performing. 

The Role of the Central Office 

T he central offices of our four study districts 
played a big role in building the data-informed 
decision-making system of which the leading indi- 
cators described in the previous section are a major 
part. They did this in three primary ways: 

• advocating for equity 
• providing supports for data-informed 
decision-making 
• establishing a data culture 

Advocating for Equity 

One of the common underlying stories that our 
respondents told us was that district leaders, partic- 
ularly the superintendents, set out early in their 
tenure to use data to highlight variation in per- 
formance among the schools in their districts. As 
one central office administrator told us: 

Within the first three weeks that [the superin- 
tendent] was here, he had developed charts of 
student data and he had disaggregated it by race 
and ethnicity and he had worked with the folks 
... in facilities planning to develop maps to 
chart out all of the results from the different 
data. And it basically showed, within the first 
three weeks that he was here, really two differ- 
ent school systems. And the results of those two 
different school systems were very different. 

A second district told a similar story. One of the 
critical indicators the district’s data office produced 
evolved from the superintendent’s mandate, at 
the beginning of his tenure in the district, that all 
the county’s larger high schools develop theme- 
based high school academies. To ensure that each 
of the new academies was recruiting diverse stu- 
dents and effectively supporting their academic 
achievements, the data office produced for each 
high school charts dubbed “circuit breakers” 


because they immediately identified areas of 
inequity, potential tracking, or disparate achieve- 
ment results based on class, race, gender, or lan- 
guage proficiency. 

The district’s data office also provides reports and 
studies that help individual high schools identify 
their challenges and opportunities. When one 
high school’s leadership team, for example, ana- 
lyzed the school’s district-provided data, it found 
that not only had the school achieved a 9.2 per- 
cent increase in its ninth-to-tenth-grade promo- 
tion rate, but that the school’s African American 
students were being promoted at an improving 
annual rate. However, the state assessment results 
suggested persistent race-based achievement gaps. 
Additionally, only twenty-six minority students 
(27 percent of the eleventh- and twelfth-grade 
minority student population) took the ACT in 
2004-2005, and minority students continued to 
be underrepresented in AP and other advanced 
courses. 

The high school used these data reports to target 
interventions, provide staff development, realign 
curricula, and increase student supports in each of 
its academies. These cultural shifts in data use and 
the impact of teaching and learning is reflected in 
what one of the district’s coaches said: 

We now break down data for each academy, 
focusing on their graduation rate and average 
GPA. Our teachers look at disaggregated data 
for race, gender, and the other NCLB cate- 
gories; we focus particularly on the percentage 
of students who go on to college and set 
improvement goals based on what the data tell 
us. 

Providing Supports for Data-informed 
Decision Making 

But telling a story with data is not enough. Each 
of our study sites provided the supports to make 
data-informed decision making a reality in the dis- 
trict by developing the technical capacity to col- 
lect information, ensure its accuracy and 
completeness, make it accessible, and present it in 
a user-friendly format. Our study districts complete 
these tasks in a number of different ways. We 
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identified several features that are common across 
our study districts, including: 

• use of data warehousing technology 

• system of standardized summative and forma- 
tive assessment 

• easy data input and interface 

• time and supports to foster data-informed dis- 
cussions 

As one central office administrator told us, 
“There’s more to it than just buying a bunch of 
equipment and software.” 

Use of Data Warehousing Technology 

To support their work, all four sites have devel- 
oped data warehouses, which link information 
stored in different locations and fomrats . 1 One 
central office administrator told us how their data 
warehouse came about: 

For years, the district has had a formative 
assessment program . . . and about a year ago, 
we needed to get better ways to get that data 
easily accessible by staff. Staff had been using 
spreadsheets and different programs, and data 
was all over the place. So we . . . started sub- 
mitting [the data to] a data warehouse. . . . 
We’re kind of in the process of putting all of 
our data together - everything from big-picture 
state assessment data to local benchmark state 
assessment data, and we’ll be bringing in more 
down-to-the -classroom-level data. The impor- 
tant part is not just collecting the data, but 
being able to present data by concept, or skill, 
or strand areas and give the teachers easy-to- 
look-at reports. 

External partners such as vendors and local educa- 
tion funds provide some sites with needed addi- 
tional capacity on technical elements and/or the 
development and implementation of processes. 
Sustainability is an issue when such expertise is 
lodged with one or two people, as is the case in at 
least one of the districts in the study. 

1 For more information on how to develop a data warehouse, see 
Mieles and Foley 2005. 


A System of Standardized Summative and 
Formative Assessments 

Each of our study districts relies heavily on state 
and local standardized tests for information about 
school and student outcomes. State tests are the 
most frequently used and manipulated to under- 
stand those outcomes, but many of our study dis- 
tricts also added local standardized-test data, 
including end-of-course exams, and districtwide 
formative assessment data. 

Having formative assessments that could measure 
student skills as they develop, rather than just at 
the end of the year or through end-of-course tests, 
was critical. One principal told us, “Until you get 
to the point where you can inform yourself about 
where your students are . . . it’s not just summative 
assessments with pass or fail, but what did you 
learn along the way.” Another principal summed 
it up: “The formative assessment piece is really 
key.” 

Having formative assessments that could be used 
districtwide required clear districtwide agreement 
on the curriculum. All four of our study districts 
have developed districtwide curricula, at least in 
the major subject areas. For example, one central 
office administrator described the impetus for 
moving to a districtwide curriculum in math: 

One of the early things we did was we did an 
audit of our math curriculum and we found out 
that we didn’t have a math curriculum. We had 
lots of different things going on and lots of 
inconsistency across the system and fifteen or 
twenty different books that we were dealing 
with. ... So the superintendent said, “We’re 
going to have this consistency across the sys- 
tem.” 

Teachers were less enthusiastic about standardized 
formative assessments, particularly in the one 
study district that included a pacing guide with its 
districtwide curriculum. 

Easy Data Input and Interface 

All the districts in our study worked to make col- 
lecting and organizing this information easy and 
included some form of classroom- or school-based 
input or scanning of some assessment data, such as 
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DIBELS reading assessments or end-of-course 
exams. They also provided an easily accessible way 
for school-based personnel to examine data about 
students or groups of students of interest to them, 
usually through a Web interface. A pair of teach- 
ers in one of our focus groups was excited to 
describe how this worked in their district: 

Teacher i: [Students] take [the test] on the 
computer. Then at the end of the day the per- 
son who’s administering uploads it to the Web 
site, and then you can go in the following day 
to the Web site. 

Teacher 2: Right. It’s really neat because ... for 
each class, you can get a printout that shows 
the scatter of kids from the lowest to the high- 
est .. . the mode and median and all that. So 
you can see the outliers so you can actually dif- 
ferentiate using that information. It shows you 
kids who maybe you thought were poor readers 
and actually they’ve got higher scores. So it’s 
only a reading test. It doesn’t test writing or 
anything and it’s on the computer, but it does 
give some valuable information. 

A principal in another district talked about the 
detailed information she receives from the central 
office. 

They break data down to the point where I can 
see kids who are behind, look at how many 
grade levels they are behind, can be broken 
down for any individual student. I just came 
here from meeting with my math teachers 
about [our formative assessments]. We have it 
broken down to the point where each faculty 
member knows what to do with each student. 

Similarly, a central office administrator walked us 
through what is available to each teacher in his 
district. 

If you’re a fourth-grade math teacher, for exam- 
ple, you get a list of your twenty-three students. 
Get their history in the district, their atten- 
dance, suspensions, strengths, and weaknesses 
on the last standardized test, projections for 
what they are likely to do without some kind of 
intervention. If you taught all four major sub- 
ject areas - math, reading, science, and social 


studies - you’d get that for each student. If you 
take that same scenario to middle school and 
you teach a block schedule, three classes of 
math, then you’d get it for those three classes. 

In another district, results are color-coded and 
“clickable”: 

So these are the fourth-grade students at [a dis- 
trict elementary school] , and it’s broken down 
by reading and by math on their [state test] 
scores. The visual of blue, green, and pink 
stands out because you want your kids to either 
be green or blue - blue meaning advanced, 
green meaning proficient, and pink meaning 
basic [or below] . If I want to sort to anything 
that’s underlined you can just simply click on 
it, and I can simply pull those kids up, and we 
can look at the kids that are scoring basic. . . . 
So, I may end up talking to the assistant prin- 
cipal about the data and results of the data, and 
I might say, “Do you have a plan for these kids 
that are right here?” And I’ll literally mention 
the kid by name . . . because we want to get to 
that level. 

This same administrator noted later in the inter- 
view that the results for each child are linked to 
an instructional planning page with suggestions 
for what to do in the classroom to address particu- 
lar academic skills. 

Time and Supports to Foster Data-lnformed 
Discussions 

Sites also have set-aside time and systematized 
processes and structures that foster conversations 
about key data. This involves training for central 
office staff, principals, and teachers to examine 
and use data and data systems, as well as regular 
data meetings and other opportunities to bench- 
mark against other classrooms, schools, and dis- 
tricts and share best practices. 

TRAINING 

The districts had multiple ways of providing train- 
ing in data use. In districts where school-based 
staff have access to a database, principals (and 
sometimes teachers) were either trained in the use 
of the database during the summer or offered 
online training. However, in one district this 
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training was only offered to central office staff and 
principals, not teachers. More often, training and 
professional development around the use of data 
was offered by districts to school-based teams and 
provided on-site by central office staff. 

The kinds of training and professional develop- 
ment around data- informed decision making that 
interviewees found most useful was when it 
included all staff and when it was embedded 
within existing groups or programs, such as a 
school-based leadership team or a principal leader- 
ship program for assistant principals. Principals in 
one district talked about data retreats for multiple 
school-based teams. As more teacher teams are 
trained in data, the use of data has become “part 
of the culture” in schools. However, as another 
principal said, 

Certain populations are more adept at using 
data. We haven’t dealt all the way down to the 
teacher population. We can do more in this 
district, and it’s probably the next step we’ll 
take. 

Another district embeds training in the use of data 
in their assistant principal development program. 
A central office administrator described the pro- 
gram: 

When you’re assigned, in your first develop- 
ment team meeting, you hit the ground run- 
ning. You are responsible for creating what’s 
called a school profile. . . . You’re basically 
doing a synopsis, if you will, of all the programs 
that are in the school. It would be like me giv- 
ing you a small booklet of everything that’s 
happening in my school, including the state of 
the union as far as the data is concerned - a 
breakdown of the data. 

There is also a specific course on data-informed 
decision making required for assistant principals. 
One principal said, 

That was really powerful because it helped you 
to do that drilling-down process and to move 
away from the blame game, but come up with 
the root causes and then possible solutions for 
it. You developed an action plan by the end of 
that class. You chose what you were going to 
tackle. That was helpful for me. 


And beyond the training in looking at and using 
data, the assistant principal development program 
fostered relationships and trust in looking at data. 
Another principal said, “You have a network of 
people to call and talk to and pose questions to.” 

REGULAR DATA MEETINGS 
Our districts also relied on “Data Chats,” “Data 
Retreats,” or some similarly named process consist- 
ing of regular meetings (annual, semiannual, or 
monthly) with school leadership teams to discuss 
school performance data. One teacher described 
the process: 

At our elementary school . . . after every [for- 
mative] assessment round ... we meet right 
after those rounds. We look at [the data] as the 
teams. Our principal has us doing data chats 
with her and the administration once a quarter. 

A principal from a different district described a 
similar process: 

[At the data retreat], the leadership team is 
there together; we’re looking at data and get- 
ting that a-ha together. . . . [We have] two days 
of rich discussion. Are we seeing results? Iden- 
tifying kids early for interventions? Are they 
making a difference? 

And an external partner from a third district in 
our study described a similar process. 

Maybe half the regions are trying to do some 
kind of cyclical process, where at some point 
early in the year they were talking about the 
problems they saw in the data, the root causes, 
coming up with some strategies they can try. 
And now they’re at the point where they are, 
in some way, sort of following up on those. Or, 
even if they didn’t start this right at the begin- 
ning of the year ... a lot of them - we gave 
them kind of a form, like a handout that they 
could use, that had plan, do, study, act. It had 
kind of the key questions. And so several of 
them after that [professional development] 
actually gave that [form] out at their meetings, 
put the data up for a particular [key indicator] 
on the screen, and said, “OK, we’re going to go 
through this ourselves together, and you’re 
going to write it down, and we are using that to 
then track going forward.” 
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BENCHMARKING AND SHARING BEST 
PRACTICES 

Our study sites frequently also took part in 
processes to collect data on classroom activities 
and student engagement, such as classroom walk- 
throughs, and quality processes, such as the 
Baldrige process or something similar that fosters 
continuous improvement. A human resources 
department in one district worked with human 
resources administrators in other districts to 
understand how their own salary, benefits, and 
other incentives for staff compared and to share 
best practices. 

Such efforts typically involve benchmarking, that 
is, providing information that helps educators 
compare their performance to similar classrooms, 
schools, or districts. As one principal described: 
The [assessment] data is there and . . . you see 
how [for example] your girls may be doing in a 
particular instructional area, as opposed to the 
boys. So, it’s broken down in many different 
ways. We look at that and we go over it and I 
review that with my team, as well, and then I 
copy it and I meet with my superintendent and 
we review it in teams [with other principals]. 

And on a team level, we’re looking at our indi- 
vidual school data, but we’re also making com- 
parisons as to how one school is doing 
compared to another school. And we discuss 
all of the targeted areas as a team, what strate- 
gies are most effective or what best practices 
are being implemented in different schools that 
you may want to take back and share with your 
staff and perhaps implement in your school. So, 
that’s a very meaningful process because you 
get a lot of ideas [with] respect to best practices 
based on the data. 

As reflected in the quote above, benchmarking 
and sharing best practices went hand in hand in 
our study districts. Study participants from each 
district talked about developing and using oppor- 
tunities to share best practices determined by 
looking at data. 

For example, I might have five teachers with 
data on the table and I see that four of them, 


the kids were really struggling in one area, 
whereas this fifth teacher, kids were doing 
really well. So the question might be for that 
teacher, “What did you do? What do you think 
helped your kids be successful here, because it 
wasn’t working somewhere else.” Maybe that 
teacher might suggest, “Well, I can demon- 
strate something - we can do some peer obser- 
vations.” She might have something real 
concrete that she can answer, because the 
answers are not always simple as, “This is what 
I did. Just do it and it’s going to work for you.” 

Another principal talked about ideas he has 

received from his team meetings with other 

schools in his district: 

Our attendance [improvement program] is 
called “Are You Present?” So, that’s where we 
take each month’s class with the highest atten- 
dance and we give them a pizza party. Well, 
that program has been very effective in my 
school and that came from a school where that 
program was going on. I just implemented it 
here. So, I get other ideas on things that other 
schools are doing that I can turn around here. 
That’s just one. Some of the ideas are not as 
clear and concrete, but it also provokes 
thought of how you may apply or connect dif- 
ferent parts of the program. 

Another principal described how having discus- 
sions about data 

allows teachers to take a look, and a hard look, 
at the data and start talking about, or having 
discussions about, “Well, what did you do in 
your class?” Especially if you have a team of 
teachers who maybe have individuals whose 
kids did really well on their tests, as opposed to 
... a teacher’s classroom who didn’t do so well 
on the test, then that fosters that kind of a dis- 
cussion. So, it helped me because I can help 
facilitate that and essentially, “Well, what did 
you do differently?” So, it’s not - it becomes 
less of a “gotcha” and more of a situation where 
it’s really bringing the best ideas, the best prac- 
tice to the table. 
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Establishing a Data Culture 

All of these efforts have helped build a culture of 
data use in our four study districts. While not 
every person in every district is a “power user” of 
data, our respondents told us that a critical mass 
has emerged in each of their districts. Data- 
infonned decision making has become a regular 
part of their practice. 

I think [the superintendent’s] cabinet doesn’t 
fear data anymore. [The research director] has 
helped them to understand that we don’t have 
to be reactionary. We have trend data, we can 
show it, and we can begin to ask better ques- 
tions about why, instead of thinking, “We have 
to get the message out that math teachers [for 
example] better start doing this.” We don’t 
panic any more. It can be about growth rather 
than reacting to any given force. We don’t 
have to walk on eggshells at a principals’ meet- 
ing; we can throw up district data by school or 
district and we can have conversations about 
it. 

Similarly, a central office administrator described 
how teachers have reacted to efforts to share data: 
My data is going to go up against another 
teacher. Some people are nervous, and some 
people are excited; [sometimes] we compare my 
first [period class] to my fourth [period class]. 
It’s not the culture to keep things a secret. . . . 
You don’t have to worry about people thinking 
that my class is poor because they’re not get- 
ting it. [Everyone] understands that it is about 
helping the kids, making connections with 
each group. It’s a very healthy process; we look 
at trends over time. One blip does not ... if 
you have a down year, you ask “Why?” If you 
have two down years and didn’t do anything, 
then, probably, shame on you. It’s a question, 
not something to freak out about. It’s all about 
“How do I get better?” 

One external partner described the “appetite” for 
data that existed in the district, which he saw as 
unique compared with other districts he had 
worked with. 


Respondents in several sites actually spoke about 
data as empowering and contributing to a sense of 
efficacy. Our informants made it clear that this 
was not just a monitoring or compliance-oriented 
function, but rather that examining data was a key 
aspect of developing a professional learning com- 
munity. 

We’re going to build a professional learning 
community. We’re going to establish a system 
where data drives our decision making and that 
we feel that we are equal partners in the deci- 
sion making, and we’re going to take a very 
close look, a transparent look, at our school 
data. We’re going to open it up for all to see so 
that we all take ownership of the problem. 

In one district this sense of data as an integral and 
powerful part of its work extended to involving 
students in keeping track of their own progress 
and using data to inform students’ efforts: 

The individual kids have data books and . . . 
you can go up to them and they’ll tell you, “You 
know, I didn’t do so good on punctuation 
today,” and you say, “Well, what’s the matter?” 
and they go, “I don’t know about the capital- 
ization,” you know, little kids. So they know 
what they’re trying to achieve. They know how 
to keep track of what they’re trying to achieve. 

- Central office administrator 

The educators in this district believe that using 
data not only improves their own practice, but 
also increases student engagement in learning. 

Central Office Self-Evaluation 

W hile our respondents were able to talk in 
depth about the indicators they used to 
understand student, teacher, and school perform- 
ance, they could provide little information about 
how they determined the effectiveness of central 
office supports, one of the central questions of the 
study. When central office staff did respond, their 
responses fell into two broad categories - internal 
evidence (from school staff) and external evidence 
(from the community or another external part- 
ner). In only one district was there any evidence 
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that this data on central office supports was used 
to make changes within the central office. 

Internal Evidence 

When central office staff talked about internal 
evidence, they cited informal evidence such as e- 
mail or conversations, as well as some more struc- 
tured data such as survey data. 

Most feedback comes as appreciation. 

When I trained teachers, I asked for feedback 
from them. I went to schools twice a year and 
gave a short survey: What do you like? What 
could be improved? You have hit on something 
that’s very weak. There’s nothing formal I have 
right now. The fact that my e-mail stays 
clogged and lots of people call me is an indica- 
tor people think I can help them. We need 
something more, in a formal sense. 

We don’t do enough of that [getting feedback 
on central office supports]. [Another assistant 
superintendent] and I started giving surveys 
and asking for feedback at our principals’ meet- 
ings. We did that for a while. Most of the time 
our principals have come so far that they give 
us honest feedback. We went away from the 
formal types of feedback. We gave some year- 
long surveys last year by topic areas and also 
asked, “What do you need? This year we asked, 
“What was our strong point this year? What do 
you need to make your job easier?” We want to 
give them what they need to help them, not 
waste their time. 

Also, one district talked about developing surveys 
for teachers and school leaders, though at the time 
it was in the strategic plan and not yet enacted. 

External Evidence 

Feedback on central offices also came from exter- 
nal partners and the community. One central 
office administrator talked about an external 
organization that led an inquiry into central office 
practice: “[The external review] revealed powerful 
infomration on how we [the central office] could 
operate. I think that move to be viewed as a sup- 
port organization was affirming.” 


Another district routinely held meetings with 
businesses and parent organizations to give feed- 
back. As part of these meetings, parents received 
trainings on assessments and data, allowing for 
“input both ways.” 

Finally, one district went beyond informal data 
and surveys and actually used data to make 
changes at the central office. One central office 
administrator said, 

I’m in the process right now of reducing 
regional superintendents. Just for economic 
factors . . . and it was done on data. It was done 
around regional’s ability to move schools. It 
was done on looking at the key performance 
indicators and the whole issue around growth 
after we got past issues around certification and 
things of that nature. . . . Data is the thing that 
helps us make those kinds of decisions. 

Overall, however, there was not a lot of evidence 
from our respondents that central office was 
addressing the quality of its supports to schools . 2 

While many schools and districts have begun the 
important process of examining data about teach- 
ers and students, few have put central office prac- 
tices under similar scrutiny. To be sure, districts 
are accountable to the community for student per- 
formance and for proper management of taxpayer 
funds. But just as with schools, these lagging meas- 
ures of performance do not say whether districts 
are putting in place the infrastructure that will 
ensure positive results for students in the coming 
months and years. They do not say whether dis- 
tricts have the capacity to support schools’ instruc- 
tional-improvement efforts, or whether they are 


2 The dearth of data might be explained by a problem with data col- 
lection. In one site, questions focused more on how knowledge is 
spread throughout the district, not on the effectiveness of central 
office supports. 
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providing the curriculum and professional devel- 
opment support schools need. As a respondent in 
one district told us: 

Interviewee: What we want to do is, we want 
to be able to say, “This AP’s leadership made a 
difference in student achievement. The lead- 
ership behaviors of this person increased stu- 
dent achievement.” We want to make that 
link. Now, that’s my goal. 

Interviewer: So, are you tracking that in some 
way? 

Interviewee: Well, we’re trying to. It’s difficult 
to track. I’m finding it’s more and more diffi- 
cult to track, but ... I can leave this with you 
- this is a draft of a survey that I expect to use 
at the end of next year to collect this data. It 
walks the AP through what they really should 
be doing with this team or department to get to 
this end result where they could say at the end, 
“Did student achievement increase, and which 
of your actions had the most impact?” 

Lessons Learned: Using Data 
to Inform District Decision 
Making 

T he four districts in our study are advanced in 
using data to inform their decision making. 

We learned a number of things from them that 
should inform the work of other districts that want 
to improve systemwide. 

•2* Though they might not be referred to as such, 
leading indicators for education exist and are 
being used to differentiate instruction and 
improve outcomes for students. 

•2* Many of the leading indicators already in use, 
such as third-grade reading proficiency and 
student age compared with credit accumula- 
tion, are data that school districts have long 
collected and are relatively easy to measure. 
But there are other indicators, such as student 
engagement and teacher quality, that are 
harder to measure and are essential to under- 
standing student success. 


*2* Our study districts have already prioritized a 
set of indicators, which we call leading indi- 
cators. Their next step however, is not just to 
let a few indicators rise to the top, but to syn- 
thesize these indicators along with other data 
they collect, so that they can create precise 
portraits of student and school achievement. 
By understanding various student “trajecto- 
ries” or “pathways” these districts hypothesize 
they can intervene earlier and more effec- 
tively to ensure that students meet their true 
potential. 

*2* School district central offices play a critical 
role in developing leading indicators as one 
part of a broader data-informed decision-mak- 
ing system. Central office leaders do this by 
advocating for equity, especially in terms of 
outcomes by race and ethnicity; by providing 
time, infrastructure, and supports that align 
all the work of the district; and, perhaps most 
important, by establishing a data culture, 
where information is sought out, discussed, 
and acted upon. 

*2* For all the emphasis on understanding school, 
student, and teacher performance, there was 
no comparable focus on measuring the effi- 
cacy of central office supports. Central office 
staff relied primarily on anecdotal evidence to 
assess whether they were adding value to the 
work of school-based educators. 

*2* Leading indicators are only one part of a 
data-informed decision-making system. In 
addition to the elements described earlier - a 
data warehouse, well-aligned and well-imple- 
mented curricula, formative and summative 
assessments, easy access to data, and support 
for using data - educators need not only good 
leading indicators but also good lagging indica- 
tors. For example, the desire of many of the 
respondents in our study districts to have 
more information about the perfomiance of 
their students in college is an effort to under- 
stand the outcomes of the education they 
provide. 
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•h Similarly, the leading indicators we’ve identi- 
fied so far are dominated by relatively easy-to- 
measure types of data and are what many 
would describe as intermediate outcomes. For 
example, depending on when the data is 
looked at and how the data is used, third- 
grade reading proficiency could be either a 
lagging or a leading indicator. As our study 
districts and others make progress in identify- 
ing student trajectories and the critical indi- 
cators along the way, understanding what is a 
leading and what is a lagging indicator will be 
less important. The critical task will be incor- 
porating harder-to-measure constructs such as 
engagement and teacher quality and imple- 
menting and building the interventions that 
will be necessary when such indicators show 
that these are lacking. 

This is an enormous challenge. Our study districts 
- which are on the cutting edge of data collection 
and use - have made big efforts to use the data 
they already have more effectively, to collect new 
data, to synthesize data and understand student 
trajectories and to act on what they learn from all 
of it. But even they struggle to find the time, the 
resources, or the expertise to also collect data on 
harder-to-measure concepts that reflect the kinds 
of rich learning environments we want our chil- 
dren to have. For them and others to do so will 
require much deeper collaboration with partners - 
higher-education institutions, community-based 
organizations, and local governments, to name 
three - through the sharing of data and resources. 
Some districts have begun that process with 
higher-education institutions, but its breadth and 
use are so far limited. This collaboration would 
more widely and deeply share accountability and 
responsibility for children throughout the commu- 
nity. 

However, there is strong interest by these districts 
in developing new indicators that would push into 
these more difficult-to-measure areas. At a con- 
vening of representatives from the four districts in 
June 2007 to review initial findings from this 
study, participants were particularly interested in 


developing tools for looking at central office prac- 
tice and program implementation and for assessing 
teacher beliefs and student engagement. Further- 
more, there is a great desire to connect districts 
that are doing advanced work in data- informed 
decision making, as well as working with less- 
developed school districts. Creating multiple safe 
spaces for districts to share challenges around data 
(or a district’s “dirty laundry,” as one district repre- 
sentative put it) is a crucial next step in the devel- 
opment of leading indicators. 
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Appendix: Research Protocols 

This appendix contains the role-specific protocols used for the interviews in our study. Most protocol 
questions are mapped to the Annenberg Institute’s Framework for Data-Informed Decision Making using 
the following notation: 

KM - Knowledge Management 
KS - Knowledge Support 
KC - Knowledge Creation 
KD - Knowledge Dissemination 


Preliminary Interview for District Research/Accountability Director 


Opening statement: Hi, my name is and I 

am part of a research team at the Annenberg Insti- 
tute for School Reform conducting a series of 
interviews with districts that are leaders in data- 
driven decision making. Your district has come up 
repeatedly as one doing innovative things with 
data-driven decision making and I’d like to ask 
you a few questions. It should take no more than 
30 minutes or so. 

1. Tell me a little about the ways your district 
uses data to inform its decision making. 

2. What data do teachers get about students 
from the district? How are teachers supported 
to use those data? 

3. What types of data do schools get from the 
district? How are school leaders supported to 
use those data? 

4. What kind of training on interpreting and uti- 
lizing data does the district provide for 

• teachers? 

• principals? 

• district administrators? 


5. In addition to student-outcome data, what 
other data does the district systematically 
collect? 

6. Can you give me an example of a way in 
which the district uses data to change the 
system by which schools are supported? 

7. Can you give me an example of a way in 
which the district learned something from 
data that led to a change in central office 
practices? 

8. Do you consider [district] a data-driven dis- 
trict? Why or why not? 

9. How do you share the district’s data with 
your stakeholders? 

10. What are the things the district is focusing 
on now to improve its data system? 

1 1 . Is there anything else we should know about 
your data-driven decision-making system? 
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Pre-visit Profile: Basic Dimensions of the Data- Informed Decision-Makins System 


The goal here is to collect information to describe 
the basic dimensions of the district’s data- 
informed decision-making system. We seek the 
basic layout of the system by getting responses to 
the following questions. Many of these may have 
already been addressed in our preliminary inter- 
view and it is not necessary to ask every respon- 
dent about all of them. 

Research teams should attempt to address these 
questions before visiting a site. The teams should 
make note of items that can’t be answered through 
preliminary interview information or other docu- 
ments and ask these questions when they get to 
the site. 

1. What are the different components of your 
data system? (KM) 

2. What technology (hardware/software) does 
your system require? (KM) 

3. What are the particular indicators/data points 
regularly collected by the system? (KM) [Make 
sure to probe specifically for implementation or 
process indicators] 


4. Who has access to the data (district 
leaders/school leaders/teachers/parents) ? 

5. How regularly are the different components 
used? (KM) 

6. What does professional development for using 
data look like in the district for the following: 

(KS) 

• District administrators 

• Principals 

• Teachers 

• Others 

7. District Context 

• Centralization of curriculum 

• Stability of superintendent 

• District demographics 

• Reform history 

• State/local context 
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General Interview for Cabinet-Level District Leaders 


Opening statement: Hi, my name is and I 

am part of an Annenberg Institute for School 
Reform research team investigating data-informed 
decision making in school districts. We did some 
preliminary work and selected just a few districts 
as innovative data users - yours among them. The 
purpose of this interview is to learn more about 
the ways that your district is using data. 

1. We have already gotten a sense of the key 

components of your data-informed decision- 
making system. They are , , and 

. Have I left anything out? 

2. There seems to be a fair amount of effort to 
analyze student impact data in your system. 

Are there other data that we haven’t talked 
about that are collected in the system? [such as 
central office customer service surveys; profes- 
sional development evaluations, etc.] 

• What other kinds of data don’t you have 
access to that would be helpful to have ? 

3. Your district seems to have made a big com- 
mitment to Data-informed Decision-Making. 
Why? (KS) 

4. What kinds of training have you received to 
use [ask for each, X, Y, Z] of the district’s data 
system? Is it sufficient? Why or why not? (KS) 

• What other kinds of training do you think 
would be helpful for you? Why? 

5. How are data used: (KS) 

• in central office meetings? 

• in your departmental meetings? 

• are there other meetings or opportunities to 
look at data? 

[Probe for why they have chosen to use data in this 
way.] 

6. Are there specific “high- leverage” indicators 
that have emerged - either for the system or 
for your department - as particularly powerful, 
useful, or predictive? (KM) 

[If asked for clarification, examples misht be num- 
ber of teaching vacancies or number of over-age 
ninth-sraders.] 


7. What other data or indicators would it be 
helpful to have access to? 

8. How is your use of data changing district 
practices? (KC) 

• Can you give me a specific example? 

[If the example is about teaching and 
learning then probe specifically for 
changes in central office; if it’s about cen- 
tral office practice then probe for teaching 
and learning.] 

• How, if at all, has this data system influ- 
enced central office practices/teaching 
and learning in the district? Can you give 
me an example? (KC) 

• How do the things that you learn get 
spread throughout the district [teachers, 
principals, schools] ? 

[Note: This is the only Organizational Learning 
question. Be sure to stress it.] 

[Probe for specific systems/mechanisms to 
spread knowledge through the system] 

9. How do you get feedback on how the data 
system is functioning and how, if at all, does 
that change the way the system works? 

10. How, if at all, is your use of data changing 
the district’s relationships with external 
stakeholders, such as parents or community 
organizations? Can you give me an example? 
[Transition: Now I’d like to ask you about how 
you use data personally.] 

1 1 . How do you personally use data to inform 
your decision-making? 

12. What’s something you have learned from 
your data that has changed how you do your 
job? (KC) 

[Going back to the ways the district uses data] 

13. What do you think are the biggest chal- 
lenges for your district in using data to 
inform decision making? 

14. Who else should we talk to in order to 
understand how data is used in this district? 


Beyond Test Scores: Leading Indicators for Education 


General Interview for District-Level System Specialists 


This interview is for specialists associated with a 
particular component of the system. 

Opening statement: Hi, my name is and I 

am part of an Annenberg Institute for School 
Reform research team investigating data-informed 
decision-making in school districts. The purpose 
of this interview is to learn more about the ways 
that your district is using data. 

I’d like to focus on [component] of the district’s sys- 
tem in this interview. 

1 . Can you give me a brief overview of how 
[component] is used in the district? (KM) 

2. Please describe the training that people get to 
use [component]? (KS) 

[Probe for:] 

• central office staff - who, frequency, and 
type of training 

• school leaders - who, frequency, and type 
of training 

• teachers - who, frequency, and type of 
training 

• others - who, frequency, and type of train- 
ing 

3. What meetings or other opportunities exist 
for people to discuss the data that come from 
[component]? (KS) 

[Note: Need to ask separately for district and 
school levels] 

4. How are data influencing practice in the dis- 
trict? (KC) 

• Can you give me a specific example? 

• [If the example is about teachins and learnins 
then probe specifically for changes in central 
office; if it’s about central office practice then 
probe for teaching and learning] How, if at all, 
has this data system influenced central 
office practices/teaching and learning in 
the district? Can you give me an example? 

(KC) 


• How do the things that you learn get 
spread throughout the district? [teachers, 
principals, schools]? 

[Note: This is the only OL question. Be sure 
to stress it. ] 

[Probe for specific systems/mechanisms to 
spread knowledge through the system.] 

5. How do you get feedback on how the data 
system is functioning and how, if at all, does 
that change the way the system works? 

6. Are there specific “high-leverage” (i.e., use- 
ful, predictive) indicators that have emerged 
- either for the system or for your depart- 
ment - as particularly powerful? (KM) 

[If asked for clarification, examples might be 
number of teaching vacancies or number of over- 
age ninth-graders.] 

7. What other kinds of data or indicators 
would it be helpful for the district to have 
access to? 

8. How, if at all, is your use of data changing 
the district’s relationships with external 
stakeholders, such as parents or community 
organizations? Can you give me an example? 

9. What do you think are the biggest chal- 
lenges for your district in using data to 
inform decision making? 

10. Who else should we speak to in order to 
understand data use in the district? 
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Interview for School -Level Users 


This interview is for principals and teachers, 
either individually or in focus groups. 

Opening statement: Hi, my name is and I 

am part of an Annenberg Institute for School 
Reform research team investigating data-informed 
decision-making in school districts. The purpose 
of this interview is to learn more about the ways 
that your district is using data. 

1. I’ve heard of these elements of the data system 
in your district: X, Y, and Z. How useful do 
you find these different components of the 
data system? 

• [Probe for each of the different components.] 

• [If respondent says he/she doesn’t find them 
useful, ask why not.] 

2. What kinds of training have you received to 
use [ask for each, X, Y, Z] of the district’s data 
system? Is it sufficient? Why or why not? (KS) 

• What other kinds of training do you think 
would be helpful? Why? 

3. If you have questions about using data, who 
can you go to for assistance? (KS) 

4. As a school community, are there other ways 
you use data regularly? Please describe, [e.3., 
data from a srade sroup formative assessment, a 
schoolwide writing prompt.] 


5. Can you give an example of how if at all, 
your instructional/leadership practices have 
been influenced by data? [Clarification if 
needed on district vs. school-collected data] 
(KC) 

6. Are there specific indicators that emerged as 
particularly powerful for your grade group, 
subject, classroom, or school? [If asked for 
clarification, examples misht be performance on 
open-ended items, or reading comprehension 
scores] 

• What other kinds of data don’t you have 
access to that would be helpful to have? 

7. How, if at all, has the data system influenced 
the ways in which you work with families 
and community groups? (KD) 

8. How, if at all, do you learn about best prac- 
tices from other teachers or schools in the 
district? (KC) 

9. Do you think the benefits of using data are 
worth the investment of effort you are 
required to make? Why or why not? 

10. What do you think are the biggest chal- 
lenges for your district in using data to 
inform decision-making? 


Beyond Test Scores: Leading Indicators for Education 


General Interview for Outside Vendors/Partners 


Opening statement: Hi, my name is and I 

am part of an Annenberg Institute for School 
Reform research team investigating data-informed 
decision-making in school districts. The purpose 
of this interview is to learn more about the ways 
that your district is using data. 

1. How long have you been working in [district]? 

2. Tell me about how you/your organization sup- 
port the use of data in [district], 

[If respondent does not mention provides training, 
probe for that. If they don’t provide training, skip 
to question 4.] 

3. Please describe the training that people get to 
use [component]? (KS) 

[Probe for:] 

• central office staff - who, frequency, and type of 
training 

• school leaders - who, frequency, and type of 
training 

• teachers - who, frequency, and type of training 

• others - who, frequency, and type of training 

4. Are there specific “high- leverage” (i.e., useful, 
predictive) indicators that have emerged as 
particularly powerful? 

[If asked for clarification, examples might be num- 
ber of teaching vacancies or number of over-age 
ninth-graders] 


5. What other kinds of data or indicators would 
it be helpful for the district to use ? 

6. How are data from [component] influencing 
practice in the district? (KC) 

• Can you give me a specific example? 

7. How do the things that are learned from 
[component] get spread throughout the dis- 
trict [teachers, principals, schools]? 

[Note: This is the only Organizational Learning 
question. Be sure to stress it.] 

[Probe: Are there specific systems/mechanisms 
to spread knowledge thru the system?] 

8. How do you get feedback on how the data 
system is functioning and how, if at all, does 
that change the way the system works? 

9. Does [district] differ in any ways they are using 
data compared to other districts that you are 
working with? How so? 

10. What do you think are the biggest challenges 
for [district] using data to inform decision- 
making? 

1 1 . Who else should we talk to in order to under- 
stand how [district] uses data to inform deci- 
sion-making? 
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